The TREC 2004 genomics track categorization task: classifying full text biomedical documents

BACKGROUND: The TREC 2004 Genomics Track focused on applying information retrieval and text mining techniques to improve the use of genomic information in biomedicine. The Genomics Track consisted of two main tasks, ad hoc retrieval and document categorization. In this paper, we describe the categorization task, which focused on the classification of full-text documents, simulating the task of curators of the Mouse Genome Informatics (MGI) system and consisting of three subtasks. One subtask of the categorization task required the triage of articles likely to have experimental evidence warranting the assignment of GO terms, while the other two subtasks were concerned with the assignment of the three top-level GO categories to each paper containing evidence for these categories. RESULTS: The track had 33 participating groups. The mean and maximum utility measure for the triage subtask was 0.3303, with a top score of 0.6512. No system was able to substantially improve results over simply using the MeSH term Mice. Analysis of significant feature overlap between the training and test sets was found to be less than expected. Sample coverage of GO terms assigned to papers in the collection was very sparse. Determining papers containing GO term evidence will likely need to be treated as separate tasks for each concept represented in GO, and therefore require much denser sampling than was available in the data sets. The annotation subtask had a mean F-measure of 0.3824, with a top score of 0.5611. The mean F-measure for the annotation plus evidence codes subtask was 0.3676, with a top score of 0.4224. Gene name recognition was found to be of benefit for this task. CONCLUSION: Automated classification of documents for GO annotation is a challenging task, as was the automated extraction of GO code hierarchies and evidence codes. However, automating these tasks would provide substantial benefit to biomedical curation, and therefore work in this area must continue. Additional experience will allow comparison and further analysis about which algorithmic features are most useful in biomedical document classification, and better understanding of the task characteristics that make automated classification feasible and useful for biomedical document curation. The TREC Genomics Track will be continuing in 2005 focusing on a wider range of triage tasks and improving results from 2004

Advancing Biomedical Image Retrieval: Development and Analysis of a Test Collection

Objective: Develop and analyze results from an image retrieval test collection. Methods: After participating research groups obtained and assessed results from their systems in the image retrieval task of Cross-Language Evaluation Forum, we assessed the results for common themes and trends. In addition to overall performance, results were analyzed on the basis of topic categories (those most amenable to visual, textual, or mixed approaches) and run categories (those employing queries entered by automated or manual means as well as those using visual, textual, or mixed indexing and retrieval methods). We also assessed results on the different topics and compared the impact of duplicate relevance judgments. Results: A total of 13 research groups participated. Analysis was limited to the best run submitted by each group in each run category. The best results were obtained by systems that combined visual and textual methods. There was substantial variation in performance across topics. Systems employing textual methods were more resilient to visually oriented topics than those using visual methods were to textually oriented topics. The primary performance measure of mean average precision (MAP) was not necessarily associated with other measures, including those possibly more pertinent to real users, such as precision at 10 or 30 images. Conclusions: We developed a test collection amenable to assessing visual and textual methods for image retrieval. Future work must focus on how varying topic and run types affect retrieval performance. Users' studies also are necessary to determine the best measures for evaluating the efficacy of image retrieval system

The MERG Suite: Tools for discovering competencies and associated learning resources

This is an Open Access article distributed under the terms of the Creative Commons Attribution Licens

A stimulus to define informatics and health information technology

<p>Abstract</p> <p>Background</p> <p>Despite the growing interest by leaders, policy makers, and others, the terminology of health information technology as well as biomedical and health informatics is poorly understood and not even agreed upon by academics and professionals in the field.</p> <p>Discussion</p> <p>The paper, presented as a Debate to encourage further discussion and disagreement, provides definitions of the major terminology used in biomedical and health informatics and health information technology. For informatics, it focuses on the words that modify the term as well as individuals who practice the discipline. Other categories of related terms are covered as well, from the associated disciplines of computer science, information technolog and health information management to the major application categories of applications used. The discussion closes with a classification of individuals who work in the largest segment of the field, namely clinical informatics.</p> <p>Summary</p> <p>The goal of presenting in Debate format is to provide a starting point for discussion to reach a documented consensus on the definition and use of these terms.</p

Enhancing access to the Bibliome: the TREC 2004 Genomics Track

BACKGROUND: The goal of the TREC Genomics Track is to improve information retrieval in the area of genomics by creating test collections that will allow researchers to improve and better understand failures of their systems. The 2004 track included an ad hoc retrieval task, simulating use of a search engine to obtain documents about biomedical topics. This paper describes the Genomics Track of the Text Retrieval Conference (TREC) 2004, a forum for evaluation of IR research systems, where retrieval in the genomics domain has recently begun to be assessed. RESULTS: A total of 27 research groups submitted 47 different runs. The most effective runs, as measured by the primary evaluation measure of mean average precision (MAP), used a combination of domain-specific and general techniques. The best MAP obtained by any run was 0.4075. Techniques that expanded queries with gene name lists as well as words from related articles had the best efficacy. However, many runs performed more poorly than a simple baseline run, indicating that careful selection of system features is essential. CONCLUSION: Various approaches to ad hoc retrieval provide a diversity of efficacy. The TREC Genomics Track and its test collection resources provide tools that allow improvement in information retrieval systems

The International Academy of Health Sciences Informatics (IAHSI): 2020 Report

Objectives: To summarize the major activities of the International Academy of Health Sciences Informatics (IAHSI) in the 2020 time period and to welcome its 2020 Class of Fellows. Method: Report from the members of the Academy's Board. Results: Due to the SARS-CoV-2 pandemic, both Plenary meetings in 2020 had to be organized as virtual meetings. Scientific discussions, focusing on mobilizing computable biomedical knowledge and on data standards and interoperability formed major parts of these meetings. A statement on the use of informatics in pandemic situations was elaborated and sent to the World Health Organization. A panel on data standards and interoperability started its work. 34 Fellows were welcomed in the 2020 Class of Fellows so that the Academy now consists of 179 members. Conclusions: There was a shift from supporting to strategic activities in the Academy's work. After having achieved organizational stability, the Academy can now focus on its strategic work and so on its main objective.S

What Makes a Top-Performing Precision Medicine Search Engine? Tracing Main System Features in a Systematic Way

From 2017 to 2019 the Text REtrieval Conference (TREC) held a challenge task on precision medicine using documents from medical publications (PubMed) and clinical trials. Despite lots of performance measurements carried out in these evaluation campaigns, the scientific community is still pretty unsure about the impact individual system features and their weights have on the overall system performance. In order to overcome this explanatory gap, we first determined optimal feature configurations using the Sequential Model-based Algorithm Configuration (SMAC) program and applied its output to a BM25-based search engine. We then ran an ablation study to systematically assess the individual contributions of relevant system features: BM25 parameters, query type and weighting schema, query expansion, stop word filtering, and keyword boosting. For evaluation, we employed the gold standard data from the three TREC-PM installments to evaluate the effectiveness of different features using the commonly shared infNDCG metric.Comment: Accepted for SIGIR2020, 10 page

Protecting healing relationships in the age of electronic health records: report from an international conference

We present findings of an international conference of diverse participants exploring the influence of electronic health records (EHRs) on the patient-practitioner relationship. Attendees united around a belief in the primacy of this relationship and the importance of undistracted attention. They explored administrative, regulatory, and financial requirements that have guided United States (US) EHR design and challenged patient-care documentation, usability, user satisfaction, interconnectivity, and data sharing. The United States experience was contrasted with those of other nations, many of which have prioritized patient-care documentation rather than billing requirements and experienced high user satisfaction. Conference participants examined educational methods to teach diverse learners effective patient-centered EHR use, including alternative models of care delivery and documentation, and explored novel ways to involve patients as healthcare partners like health-data uploading, chart co-creation, shared practitioner notes, applications, and telehealth. Future best practices must preserve human relationships, while building an effective patient-practitioner (or team)-EHR triad

Clinical outcomes resulting from telemedicine interventions: a systematic review

BACKGROUND: The use of telemedicine is growing, but its efficacy for achieving comparable or improved clinical outcomes has not been established in many medical specialties. The objective of this systematic review was to evaluate the efficacy of telemedicine interventions for health outcomes in two classes of application: home-based and office/hospital-based. METHODS: Data sources for the study included deports of studies from the MEDLINE, EMBASE, CINAHL, and HealthSTAR databases; searching of bibliographies of review and other articles; and consultation of printed resources as well as investigators in the field. We included studies that were relevant to at least one of the two classes of telemedicine and addressed the assessment of efficacy for clinical outcomes with data of reported results. We excluded studies where the service did not historically require face-to-face encounters (e.g., radiology or pathology diagnosis). All included articles were abstracted and graded for quality and direction of the evidence. RESULTS: A total of 25 articles met inclusion criteria and were assessed. The strongest evidence for the efficacy of telemedicine in clinical outcomes comes from home-based telemedicine in the areas of chronic disease management, hypertension, and AIDS. The value of home glucose monitoring in diabetes mellitus is conflicting. There is also reasonable evidence that telemedicine is comparable to face-to-face care in emergency medicine and is beneficial in surgical and neonatal intensive care units as well as patient transfer in neurosurgery. CONCLUSIONS: Despite the widespread use of telemedicine in virtually all major areas of health care, evidence concerning the benefits of its use exists in only a small number of them. Further randomized controlled trials must be done to determine where its use is most effective